f_lynx.python.stuff

Strike three!

2009-06-03T01:10:00.001+04:00

I'm getting tired from how some people tend to do business in this country.

So, from now on, every single project I lead there will be no excessive tolerance to incompetence; here's how it is, from now on:

The three strike rule:

A person gets three and only three chances: the first is when he/she gets the job, then there are two possible misbehaviors (missed deadlines without a reason, recklessness, quality below ones level, ... etc.) and the person of off the project, period.

The evolution of the programmer...

2009-01-07T17:17:00.001+03:00

A little story/joke I tell my students to explain the good and bad motivation they will face and to teach them where to look first:

There is a term in Russian programming culture called the "programmer laziness" (программистская лень) and it is a good thing, if one knows how to control it, that is.

In general, it is exhibited by spending hours to automate the solving of a problem that would otherwise take five minutes to fix by hand. When this problem arises again most programmers will again dig in for hours but, some more advanced people will remember that they've already did it, will blow the dust off their old code, adjust it and run it again spending less time than it took to code in the first place but, still substantially more than it would take to do the work by hand. In turn, most of these people will repeat the process again and again every time they face the task, evolving their code. But, a fraction will notice a routine and write a problem detector virtually eliminating the time spent on even considering the task (effectively, again, taking them to a new level of abstraction) only until a sufficiently different problem is encountered and then the process needs to be repeated from the start. And so on.

That was the first evolutionary level.

People who either have evolved to the second level or were born into it will say: "Oh! now, I'm seeing this too often, I wonder what is causing it?" as soon as they see an error for the second time at max.

a new tool! :)

2008-11-14T23:30:00.001+03:00

http://gist.github.com/ ...quite useful :)

it does this:

a very nice way to host and publish code!

via

web and computers...

2008-09-20T22:54:00.001+04:00

One interesting tendency about the web these days is dramatic browser tech advancement.... just a mad idea, could this lead to practical self-applicability and/or developing the whole system in (or rather as) a browser?

And I do not mean things like Dan Ingalls' Lively Kernel, no. The idea is more of a standalone bare browser/server on hardware that can be useful both connected or not. this could include a proxy cache as store; booting from the web or local server (loading data to the cache), working with the data locally and posting changes when the system is on-line.... redefining the personal computer concept in terms generally associated with the web.

Some food for thought :)

OOP and framework cooperative reuse...

2008-09-07T05:38:00.001+04:00

I've been hitting this problem for a couple of years now...

here it is, in short: we have an algorithm or a set of algorithms that operate on one or several "terms", this is usually called a framework and implemented as a relatively monolithic (usually but not necessarily) entity with several customization points, and is used by replacing/customizing these points to achieve a variation of the provided functionality. there are numerous examples of this approach among which are the Python's logging package and the Twisted framework, to name a couple of big ones, on the smaller scale there are things like UserDict, ...etc. this approach works quite well but has two rather big weaknesses:

the bigger things get the more unnecessary complexity of design and implementation there are and in some (if not most) cases this also relates to complexity of use...

a framework is extremely hard to to implement in a way for it to be reused several times within one context...

the second point may need a little explaining. let us consider this example: we have several frameworks that simplify pythons dict creation and use, these usually require the user to define only tree or four special methods and implement the rest of the dict interface on top of that... quite straight forward, indeed. so to demo the above concept let's try and implement an object that acts as a dict and at the same time responds to a special attribute interface, both protocols access different data stores and are essentially independent, but are almost identical in their base functionality.

so some of the traditional approaches to solve this are:

implement everything by hand.

as far as my experience goes this appears to be one of the most popular ways to solve things. this is quite boring and bad so I'll skip the details and code example.

extend and wrap (the OOP way).

In other words make the object a dict-like by extending a special framework and proxy the attr access to a nested container (or vice versa).

here is an example:


import pli.pattern.proxy.utils as putils
import pli.pattern.mixin.mapping as mapping

# mapping.Mapping is quite similar to UserDict form the standard library but is quite a bit more 
# flexible. though, for this example it makes almost no difference except for the base methods
# needed by each library are a but different.
class DictLikeWithAttrs(mapping.Mapping):
    _dict_data = None
    # to add some fun, all of the instances have the same "base" state, yet still can store private 
    # data in their local namespace...
    _attr_data = {}

    def __init__(self):
        self._dict_data = {}

    # proxy the dict interface...
    putils.proxymethods((
            '__getitem__',
            '__setitem__',
            '__delitem__',
            '__iter__',
        ), '_dict_data')

    # proxy the attr methods to the dict interface of the _attr_data...
    putils.proxymethods((
            ('__getattr__', '__getitem__'),
            ('__delattr__', '__delitem__'),
        ), '_attr_data')
    # and we need to take special care with __setattr__ to avoid infinite recursion...
    def __setattr__(self, name, val):
        if name in ('_attr_data', '_dict_data'):
            return super(DictLikeWithAttrs, self).__setattr__(name, val)
        self._attr_data[name] = val

    # this will set local attrs that have priority over the global state...
    putils.proxymethod(
        ('setlocal', '__setitem__'), '__dict__')

this approach has both it's advantages:

involves far less work than the first approach.

also is simpler, if proper tools and discipline are used.

and disadvantages:

the result is not too reusable or configurable.

not simple enough (IMHO).

scales badly -- we essentially have to proxy all the methods by hand.

another reason why this scales badly is that you can extend only once per protocol and for each consecutive use you need to nest.

just imagine how would an object that supports three or four different mapping-like interfaces look like...

write an interface factory.

let's go straight to an example:


def mappingproxy(target, get='__getitem__', set='__setitem__', delete='__delitem__', iter='__iter__', depth=1):
    map = ()
    # these will enable us to disable some methods...
    if get != None:
        map += ((get, '__getitem__'),)
    if set != None:
        map += ((set, '__setitem__'),)
    if delete != None:
        map += ((delete, '__delitem__'),)
    if iter != None:
        map += ((iter, '__iter__'),)
    # sanity check...
    if map == ():
        raise TypeError, 'to generate a proxy we need at least one methods enabled.'
    # now generate the proxies...
    putils.proxymethods(map, target, 
        # this sets the namespace depth of the caller... (an off-topic, but in case you wounder :) )
        depth=depth+1)

def attrproxy(target, get='__getattr__', set='__setattr__', delete='__delattr__', iter=None, depth=1):
    return mappingproxy(target, get=get, set=set, delete=delete, iter=iter, depth=depth+1)

so our class will look like this... (the changes are in italic)


class DictLikeWithAttrs2(mapping.Mapping):
    _dict_data = None
    # to add some fun, all of the instances have the same "base" state, yet still can store private data...
    _attr_data = {}

    def __init__(self):
        self._dict_data = {}

    mappingproxy('_dict_data')

    attrproxy('_attr_data', set=None)
    # and we need to take special care with __setattr__ to avoid infinite recursion...
    def __setattr__(self, name, val):
        if name in ('_attr_data', '_dict_data'):
            return super(DictLikeWithAttrs2, self).__setattr__(name, val)
        self._attr_data[name] = val

    # this will set local attrs that have priority over the global state...
    putils.proxymethod(
        ('setlocal', '__setitem__'), '__dict__')

this is a good approach, we made the user code both shorter and more understandable (be it at an expense of adding some more complexity to the library). but, on the down side, our factory and generated code are rather static.

in essence, by automating part of the process we made the whole thing less flexible.

design the interface "stackable" form the ground up.

this is really bad as it makes everything dramatically more complex by adding dispatching to everything and confusing option and data registries/dicts/lists...

macros or templates.

I'd avoid this at this point as I'm investigating a possibility of a clean OOP approach.

I regularly use the second and third approaches but, I'm looking for something that would feel more natural...

In my view the best way to go would be defining a framework in a way that would isolate the algorithms from the terms they use, making the later overloadable.
On the small scale this is the same as a simple function, where the functionality is black-boxed, the user just inputs the arguments and gets the result; but this does not scale to collections of functions or frameworks with ease. I can't seem to find anything that would fit this in any of the systems/languages I know of (but I could be missing a thing or two).

Any Ideas? :)

P.S. you can find pli here...

the simplest design is allot more fun!

2008-08-27T21:24:00.001+04:00

The first time I noticed this, I got quite surprised... the best solution for a given problem is the simplest, most straight forward, least thrilling and usually generates remarks like "this is textbook stuff, too simple and uninteresting" and, in fact, that it may well be!
but as a side-effect of simplicity one gets tiny code base and flexibility unrestrained by bells and whistles. The result is as complex as the task it solves without any extra automation and framework magic :)

Yes, you get less of that Perl-ish "I've written this but, even I do not understand how and why it works, but it does!" thrill and none of the satisfaction it brings...

but getting the job done in a fraction of the time and watching the client try to rip it apart and reject it as being too simple, too abstract or too optimistic and not being able to do so objectively, is allot more fun ;)

GPL?

2008-08-25T03:38:00.001+04:00

I do understand the motivation and the cause that Mr. Stallman is fighting for here (and I'm with him on this!) but, somehow I do not feel comfortable forcing people to be free...

...so, I'm sticking with free free licenses for most of my work.

plus, "forcing to be free" sounds quite familiar!

P.S. this is actually a tad late response to his visit to moscow this year :)

it's time to revive this blog :)

2008-07-30T16:02:00.001+04:00

I've not been writing on this topic for quite sime time. I'll try to make myself do this on a more regular basis now...

redesign and rewrite is good!

2008-03-26T13:34:00.003+03:00

One funny thing I've noticed over the years is the more you get attached to a particular design the less you notice it's drawbacks and the more you like it, regardless of either quality or applicability to a particular purpose.

A way to avoid this set of problems I've thought of a couple of years back is to not to like any design by default and as soon as it starts to annoy do a redesign or a rewrite.

So here are the preliminary results:
+ after a couple of rewrites one instinctively motivated to write small and clear code.
+ each consecutive rewrite/redesign is faster and faster.
+ the designer gets a deeper and more practical view of the problem.
+ good for teaching design.
+ after a couple of cycles there is a sufficient codebase to combine the strongs of several approaches without a major rewrite.
+ forces a certain concise and flexible design style.
- this approach (in it's current state) is not suited for teems with varied experience levels.
- may not be deterministic at first.

One other thing I've noticed is that at first it takes more time that the usual approaches, but at the later stages there is virtually no time spent on hacking design bugs out and fixing all that depended on them in a crunch.

Agile is a similar set of approaches, but the difference here is that on each stage there is mainly a forced redesign and rewrite, targeting solving the problem differently and not reusing code.

The @decorator flaws...

2004-08-06T14:49:00.000+04:00

First of all I must state that I agree with all who think this is un-pythonic.
but considering the facts, I can not do anything about it.... thus to the purpose of this post:

If you go for it, go all the way!!!

I just set up Py2.4a2 and found out the following (can't say I am surprised):

- you can not decorate anything but a function!!

this is a shame, as it would be logical to expect this to work for the following cases:
1) variables/attributes
Example:

class A(object):

    @readonly

    someattr = 0

@register_in_store

obj = SomeClass()

2) classes
Example:

@prepare

class A(object):

    someattr = 0

each and every of the above cases currently generates a syntax error.

P.S. I'm thinking of posting this as a bug or feature request (but it needs more thought before I do)....

here is the problem in more detail...

2004-07-20T12:22:00.000+04:00

It is the same as with "super(self.__class__, self).meth()", that is, as soon as you change, or combine metaclasses for some child class, you'll get wrong method resolution, and most likely fall into infinite recursion... (see example below).

the simplest solution would be to pass the metaclass to super explicitly, but this is even worse than passing the class explicitly in the general case, as the class and the metaclass can be located in different modules/packages... etc. I can't say that this will contribute to code clarity :)

example:


 

import metaclass_example

 

 

 

class _OtherMetaClass(type):

    '''

    '''

    def meth(cls):

        print 'this is the other metaclass method!!'

        # here super is in a try block so as to make the method usable

        # as is (not only from class combinations).

        #                                  ....a terminated super! :))

        try:

            super(cls.__class__, cls).meth()

        except AttributeError:

            pass

 

class SomeUsefullClass(object):

    '''

    '''

    __metaclass__ = _OtherMetaClass

 

 

# NOTE: here we inherit from two classes which are of different types

#       or metaclasses, thus creating a metaclass conflict.

#       to resolve this conflict we need to "combine" the metaclasses

#       of the parent classes....

class _CombinationMetaclass(_OtherMetaClass, metaclass_example._MetaClass):

    '''

    '''

    pass

 

# and here is an innocent looking result.... :)

class KillerClass(metaclass_example.SomeOtherClass, SomeUsefullClass):

    '''

    '''

    __metaclass__ = _CombinationMetaclass

 

    def meth(cls):

        '''

        '''

        print 'hi from the killer :)'

        super(cls.__class__, cls).meth()

 

 

if __name__ == '__main__':

 

    SomeUsefullClass.meth()

 

    killer = KillerClass()

 

    print 'BANG!!!'

    killer.meth()

 

 

# vim: set ts=4 sw=4 nowrap :

metaclass_example_killer.py

P.S. I made a little typo in the module name in the main post (the word example was lacking the "p" :) ), now this is corrected...

this was originaly posted here.

ugly super class-method call...

2004-07-18T00:38:00.000+04:00

Yesterday I stumbled on a funny problem :)

Task:
call the original class-method extended by another class-method in a subclass.

Logical Answer:
use the super function.

The Problem:
considering that the way "super" is called, is not the most beautiful feature of the Python language (this is sed to be a wart by some). this situation created a code far uglier. to be honest, I hate this solution but I can't seem to come up with a better one... does anyone have any suggestions?

the problem is that there are situations (though quite rare, or even unlikely) when this code will break!

see the example below for more details...

a full example:


 

class SomeBaseClass(object):

    '''

    '''

    def meth(self):

        '''

        '''

        print '###', self, self.__class__

 

 

class _MetaClass(SomeBaseClass, type):

    '''

    '''

    def __init__(cls, name, bases, ns):

        '''

        '''

        if 'meth' in ns:

            setattr(cls, 'meth', classmethod(ns['meth']))

        super(_MetaClass, cls).__init__(name, bases, ns)

 

 

class SomeClass(object):

    '''

    '''

    __metaclass__ = _MetaClass

 

 

class SomeOtherClass(object):

    '''

    '''

    __metaclass__ = _MetaClass

 

    def meth(cls):

        '''

        '''

        print 'this is a class-method...'

        # and here is the problematic line of code:

        super(cls.__class__, cls).meth() # or super(cls.__metaclass__, cls).meth()

 

 

if __name__ == '__main__':

 

    basic_obj = SomeBaseClass()

    some_other_obj = SomeOtherClass()

 

    basic_obj.meth()

    SomeClass.meth()

    some_other_obj.meth()

 

# vim: set ts=4 sw=4 nowrap :

metaclass_examle.py

if anyone is interested, this is a simplified version of the structure of a module (pli.event.event or a shorthand pli.event), part of the pli library.

this was originaly posted here.

The most promessing python projects IMHO

2004-07-17T22:48:00.000+04:00

Psyco
Pyrex
PyPy